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ABSTRACT 

The importance of a state role in assessing and 
improving undergraduate education and possible initiatives that 
states can pursue are discussed. Two roles for states are identified; 
(1) state regulatory and funding mechanisms should create an 
appropriate climate and concrete incentives for inducing 
institutional self-improvement; and (2) state governments should 
monitor the performance of the state's higher education system as a 
whole by periodically collecting appropriate measures of 
effectiveness. Because of diversity among colleges and students, 
multiple means for assessing the effectiveness of both the whole 
system and individual colleges are needed. Important issues in 
determining the effectiveness of a state's college instructional 
system are: changes in student learning and development resulting 
from instruction; monitoring access to higher education; how student 
learning and develoi»nent contribute to behavior at work, the 
marketplace, and the community; and costs of programs and 
improvements. Examples of instructional ioiprovement programs at six 
colleges are described. Broad directions for state policy are 
addressed that concern issues such as incentives for institutional 
improvement and the use of quantitative information. (SW) 
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ievers for Change: 
The Role of State Government in Improving the Quality 
of Postsecondary Education 

In recent months, discussions about the quality of America's colleges and 
universities have become increasingly salient, both on and off college 
campuses. Many of these discussions have been stimulated by parallel and far 
deeper public concerns with effectiveness in elementary and secondary 
education. Many have also been stimulated by the fact that postseconaary 
education has become bewilderingly complex. The result of both these trends 
has been a growing number of critical public assessments of higher 
education— for example. Involvement in Learning , the report of the National 
Institute for Education's Study Group on the C^iditions of Excellence in 
American Higher Education, Integrity in the College Curriculum , a report of the 
American Association of Colleges, and To Reclaim a Legacy , a report of the 
National Endowment for the Humanities. 

Any historical review will reveal American higher education to be strongly 
characterized by such periods of self-examination. Why should state 
governments get involved? In answer to this question, the purposes of this 
paper are two: {!) to argue for a growing need for a state role in assessing 
and improving undergraduate education, and (2) to discuss a number of possible 
initiatives which state governments can undertake to induce positive change. 
The case for need rests on the considerable investment most states have made in 
their systems of higher education, and upon demonstrable connections between 
the effectiveness of such systems and the fulfillment of other state 
objectives. The approac'ies discussed are based on a growing body of experience 
with actual quality improvement programs in colleges and universities, and on 



the experience of several stcites which have been experimenting with ways to 
Induce and reward such programs. 

Two important points undergird both these discussions. The first is that 
higher education is a distinctive undertaking, distinguishable both in terms of 
its governance structure and in terms of the problems it faces. Despite some 
common roots, the problems of higher education and elementary/secondary 
education are quite different. Unlike the elementary/secondary world, there is 
no lack of basic talent in college classrooms. Nor are colleges and 
universit-^es for the most part dealing with student populations so deficient in 
basic skills as to render them essentially ungovernable or unteachable. 
Instead, the problems of higher education have largely been those of 
establishing instructional improvement as a real priority, of changing 
organizational structures to facilitate improvement, and of providing clear 
incentives for needed change. 

State government has a role to play in addressing these problems, but tne 
nature of higher education makes this role a special one. Historically, 
colleges and universities have been decentralized and largely self-governing 
enterprises. Their major organizational strength has been their ability to 
provide a stable environment for individual experiment and innovation— in both 
research and teaching. T'-e* principles of individual faculty authority and of 
academic freedom provide a foundation for effectiveness in these primary 
missions by ensuring that contemplated changes are reviewed and approved from a 
perspective removed from both the marketplace and from the shifting demands of 
public opinion. 
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As organizations, this means that colleges and universities have considerable 
capacity for self-improvement. But it also means that the changes which 
ultimately stick must come from within. The result can be a paradox for 
external authority. On the one hand, if higher education is left entirely to 
itself, the probable result will be a neglect of socially important tasks. On 
the other hand, if state regulatory authority is applied directly, the very 
mechanism for effectively achieving these tasks may be threatened. As a 
result, unlike reform in elementary and secondary education, the state role in 
improving the quality of higner education will be more indirect and 
circumscribed. The key is to develop policy mechanisms which trigger 
institution-level efforts toward se"!f -improvement. Examples and discussions of 
such mechanisms will form a considerable part of later sections of this paper. 

The independent nature of college and university governance also leads to a 
second important point: a state's objectives for its syste m of higher education 
should be broader than the sum of the objectives of the particular institutions 
which comprise it. When states and institutions seek "quality", they are often 
speaking of different things. For many institutions, to attain quality may 
mean to attain the research productivity, student selectivity, and national 
peer status of a "flagship" research university. But a collection of 
"flagships" may jll-s^erve state needs for access, for broad-gauged talent 
development, and for practical, applicable regional research and service. 

This means that any attempt to attain quality must carefully define and monitor 
systemic as well as institutional effectiveness. Ir^deed, attaining system 
objectives may imply substantial differentiation among insti^tutions. And it 
may mean establishing quite different templates against which to evaluate 
institutional effectiveness. Some possible dimensions of systemic 



effectiveness will be the topic of a later section of this paper. For the 
moment, however, it is important to stress that a systemic perspective argues 
against change policies which will i-nduce institutions to automatically become 
more like one another. Such policies as uniform minimum admissions standards 
and common capstone examinations may well increase "quality" levels at 
individual institutions. The question for state decisionmakers is what 
consequences such policies have for the attainment of a highly differentiated 
set of objectives. 

Taken together, these two points set an important context for defining state 
policy. From within this context, two distinct roles for state government 
emerge: 

• State regulatory and funding mechanisms should create an appropriate 
climate and a set of concrete incentives for inducing institution al 
self-improvement . This implies that the role of state government is 
important, but as a stimulant toward, rather than as a direct 
participant in, institutional change. It also implies that states 
should use all appropriate levers in concert to induce change. Funding 
mechanisms, regulations and reporting requirements, financial aid and 
other access mechanisms, and mission or program review policies all 
should be consistent in supporting such efforts. 

• State governments should monitor the performance of the state's higher 
education system as a whole by collecting appropriate measures of 
effectiveness at periodic intervals . This implies that states must 
clearly define the systemic objectives which they are trying to 
achieve, and should clearly articulate these objectives through the 
types of data which they periodically collect. It also implies that 



accountability for effectiveness rests with the system as a whole as 
much as with the individual institutions which comprise it. 

These two roles are distinct, but are mutually supporting. Concrete, reliable, 
regularly collected information on system effectiveness is needed to guide 
overall policy development aimed at institutional self- improvement. The actual 
information collected, and the form in which it is required, will also signal 
to institutions the priorities the state considers important and intends to 
pursue. At the same time, state incentives for institutional improvement, 
based upon such information, should be consistent with and preserve existing 
institutional governance arrangements, but should hold institutions accountable 
for ensuring that efforts for self-improvement consistent with state priorities 
are indeed taking place. 

1- Why Is Higher Education's Effectiveness an Issue and Why Should Sta te 
Governments Have a Role in Addressing It? 

The case for state involvanent in improving the effectiveness of colleges 
and universities rests on several foundations. One is the growing evidence 
of important connections between higher education and regional economic 
development. In the development of high technology industries, for 
example, states are increasingly recognizing that the presence of a strong 
university research and teaching base is a significant asset (Matthews and 
Norgaard, 1984). At the same time, industries and employers of all kinds 
have increasingly made known their needs for new employees with appropriate 
general skill levels, and for access to professional development 
opportunities for those already employed. A second historic foundation of 
state involvement is access. Participation in higher education brings with 
it economic and social benefits, and an important public responsibility is 
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to ensure that access to these benefits is equitable and provides maximum 
opportunities to develop individual talents. 

These twin foundations tiave provided a rationale for state support of 
higher education since the creation of the land grant institutions in the 
mid-ninteenth century. Today, however, state support underlies a vast and 
coir, lex array of institutions. Since 1950, the total number of higher 
education institutions has increased by over 60%, and the vast majority of 
this expansion is accounted for by public institutions (NIE, 1984). 
Particularly impressive has been the growth of publicly supported two-year 
colleges, which now number over 900 and which enroll about a third of these 
attending. State-supported institutions currently enroll over six and 
one-half million full-time-equivalent students— more than three-quarters of 
those attending colleges and universities (McCoy and Halstead, 1984). 
Types of institutions range from complex "multi-versities" with national 
research standing and high selectivity to small, rural public two-year 
colleges with enrollments of less than 500. Between these extremes are 
institutions as diverse as former teachers colleges aspiring to be 
comprehensive regional institutions, specialized professional and technical 
training institutes, and diverse, multi-campus urban community colleges. 

As the numbeV- and complexion of institutions have changed, so have the 
characteristics of students. Enrollment in all postsecondary institutions 
has quadrupled in the last 35 years, and with expansion has come a 
corresponding diversity. Over half of currently enrolled students are 
women, more than 15% are minorities, and more than 40% are over 25 years 
old (NIE, 1984). Demographic diversity has meant divergence from typical 
patterns of attendance: over 40% of current students attend part-time. 



almost half commute to college, over 30% of new fresl^.-nen have delayed their 
entry into college at least a year since graduating from high school, and 
more than half those seeking a Baccalaureate degree take more than the 
traditional four years to complete it (NIE, 1984). These tendencies are 
national averages; they are correspondingly greater for state-supported 
institutions. 

Part of this shift has also been due to parallel changes in higher 
education's secondary cl ientele— industries, employers, and professions. 
Advances in technology (in traditional industries as well as in new 
technology-based industries) have meant considerable expansion in the array 
of specialized occupations. In the current high-technology sector, for 
example, it is estimated that over two-thirds of the work force requires 
college training, and that over half of the degrees required are in a range 
of newly created specialized vields of study (Botkin, Dimanescu and Stata, 
1982). Colleges and universities have responded to these needs by creating 
a widening array of new disciplines. The Council on Postsecondary 
Accreditation, for example, currently recognizes and certifies programs in 
37 areas of professional training ranging from Medical Technology to Social 
Work. At present, American colleges and universities offer programs in 
more than 1100 distinct fields, more than half of them occupational. And 
Students are taking such programs in growing numbers. The traditional arts 
and sciences baccalaureate program, which enrolled over half the nation's 
undergraduate students 20 years ago, now accounts for only a third (NIE, 
1984). 

Diversity in itself, of course, is not a major cause for concern. Concern 
arises only if public objectives for higher education are not being met. 
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But diversity does call attention to the need for clarity in these 
objectives. One reason for recent attention toward higher education at the 
state level, indeed, is prompted by a straightforward desire on the part of 
those who support the enterprise to understand exactly what it is supposed 
to do. 

Even in the absence of such understanding, there is growing evidence that 
higher education may not effectively be doing some of the things that it 
should be doing. In parallel with declines in test scores among high 
school graduates, for example, have been declines in 11 of 15 major subject 
areas on the Graduate Record Examination administered to many students at 
the end of a four-year college program (NIE, 1984). National figures on 
program completion indicate that only about 60% of those who begin college 
expecting to attain a baccalaureate degree eventually complete this goal 
(Beal and Noel,d 1980). Clearly much of this performance can be attributed 
to changes in the college-going clientele, and to le poor preparation many 
students receive in high school. But some of the responsibility surely 
rests with colleges and universities themselves. 

In some ways more revealing than declines in student performance have been 
the responses of employers and professional groups to perceived 
deficiencies in the training provided by colleges and universities. More 
and more are investing in their own training programs rather than turning 
to their preferred alternative— area colleges and universities— to provide 
basic training (Matthews and Norgaard, 1984). 

The case for public involvement in higher education continues to rest, as 
it always has, on the attainment of objectives for manpower development and 
individual mobility through the development of talent. But the 



organization of public higher education places most responsibility and 
influence in the hands of the state rather than in the hands of federal or 
local authorities. This is a sHuation quite unlike that of elementary and 
secondary education. Not only do states have much to lose if undergraduate 
ba- (C skills are not being acquired by their citizens, but they also have 
the most leverage available for inducing concrete improvement through 
funding mechanisms, through reporting and accountability procedures, and 
through direct regulation. But in order for any of these mechanisms to be 
applied effectively, more basic understanding is required of exactly what 
is to be accomplished. 

2- What Does C ollege and University Effectiveness Actually Mean a nd How Can it 
Be Assessed? - 

Higher education has many purposes, and these different purposes are often 
embodied in different types of institutions. Indeed, as emphasized 
previously, state policy should recognize institutional diversity as 
healthy so long as the total pattern of outcomes produced is consistent 
with statewide educational needs. Diversity also implies that multiple 
means for assessing the effectiveness of both the system as a whole and of 
individual institutions need to be employed. But along what dimensions 
should such an assessment be structured? The state perspective demands 
that patterns of outcomes across institutions be monitored and addressed 
through policies which assign distinct missions to different institutions. 
At the same time, accountability mechanisms for individual institutions 
demand that some attempt be made to sort out many possible outcomes, to 
establish priority among them, and if feasible to directly measure them. 
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In brief, determining the effectiveness of a state's higher education 
instructional system ought to involve getting answers to four distinct 
kinds of questions: 

a. What broad changes in student learning and development have actually 
occurred in particul a r institutions as a result of instruction? At 
bottom, education is a change concept and ought to be recognized as 
such. Students enter colleges and universities with certain basic 
abilities and have developed these abilities to varying degrees based 
upon their past training and experience. Assessing the effectiveness 
of college thus means more than simply looking at what students know 
and can do at the end of the experience. It means first taking into 
account prior knowledge and characteristics (Astin, 1977). Very few 
institutional assessment programs currently meet this standard, but 
their number is growing. State policy should encourage development of 
such programs by any means appropriate. 

Most scholars currently recognize a number of distinct outcomes 
dimensions for higher education (Ewell, 1984). Individual institutions 
may legitimately vary in the priority which they accord each of these 
dimensions, but all represent important developmental outcomes of the 
college or university experience: 

Cognitive Development . Changes in actual knowledge are among the 
most easily identified college and university outcomes, but are 
among the most difficult to consistently measure. Most observers 
make a distinction between general knowledge (the presumed common 
cognitive content of the baccalaureate degree, for example) and the 
more specialized knowledge associated with particular academic 
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disciplines. Both are being increasingly assessed through concrete 
measurement procedures. Common mechanisms for examining gains in 
general knowledge inclucte the frequently administered SAT and ACT 
entrance examinations, and the verbal and quantitative portions of 
the Graduate Record Examination (GRE). Recently, specialized 
general education assessment instruments such as the ACT College 
Outcomes Measures Project (COMP) have been administered at a 
significant number of institutions. In some cases, these 
examinations have been adminis^.ered in a test-retest format in 
order to assess growth. Assessment of knov/ledge in a specific 
field often rests upon such standardized tests as the GRE Field 
Examinations currently offered in seventeen distinct fields. Other 
institutions have used such examinations as the CLEP (available for 
over 30 fields) to assess specialized course-level knowledge. 
Still others have successfully developed their own examinations. 

Skills Development . Most of the recent concern with educational 
effectiveness has not been about what students know but about what 
they can or cannot do. Changes in levels of performable skills 
also represent an important category of outcomes. Like cognitive 
deve^lopment, however, skills development may be pursued along two 
important sub-dimensions. General functioning skills include such 
broad abilities as writing, speaking, and mathematical ability. 
They also include such higher-order skills as the ability to think 
critically, to organize inquiry, and to assess information. 
Instruments such as the ACT-COMP, which require actual student 
performance in response to diverse stimuli, represent good 
assessments of these abilities, and several institutions have 
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successfully employed specialized tests of general skills both 
diagnostically and to evaluate the curriculum. For the assessment 
of specialized, often job-related, skills, established standards of 
performance are generally available through the certification 
process. Many such programs require demonstrations of actual 
performance in real or simulated job situations. 

Attitudinal Development . A final dimension of change as a result 
of the college experience involves the development of student 
attitudes and values. A primary traditional argument for higher 
education is its potential to civilize— to inculcate basic values 
of knowledge-seeking, tolerance for different perspectives, or 
development of self-worth. For many institutions, particularly 
private colleges, development of such attributes is seen as a 
primary goal. Academic researchers have long concentrated their 
efforts on investigating such changes (for example, Chickering, 
1969; Astin, 1977). Currently many instruments for assessing 
attitudinal development are available for and in use by colleges 
and universities. Among them are the ACE/CIRP, ACT-Evaluation 
Survey Service, and the College Board/NCHEMS Student Outcomes 
Information Service (SOIS). Many other institutions have 
constructed their own such instruments and administer them 
regularly. 

From the state perspective, changes in student learning and development 
represent means to an end. Knowledge for its own sake is a noble goal, 
but one not immediately arguable as a public priority unless tied 
directly to the development of identifiable skills or changes in actual 
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behpvior as manifested in the workplace and the community. For this 
reason, the state role in directly monitoring cognitive or 
developmental outcomes shou.ld probably be limited. A significant 
potential exception is in the attainment of basic skills. Here, it may 
be important for states to monitor a minimum level of acceptable 
performance as an indicator of systemic effectiveness. The same 
argument can be made for rates of passage on established specialized 
certificp.tion examinations. 

At the institutional level, on the other hand, regular assessment of 
cognitive growth and attitudinal development is critical. Only through 
such a process can curricular effectiveness be judged or individual 
student progress properly monitored and directed. Despite its 
importance at the institutional level, however, most colleges and 
universities remain unsystematic in the ways they address cognitive 
assessment. As noted later, in cases where this has been given 
considerable attention, the payoffs in instructional effectiveness have 
been considerable. 

For Whom ? Colleges and universities may be demonstrably effective in 
producing changes on all of the above dimensions and still fall short 
of state objectives. This will be the case if it is clear that access 
to these benefits is limited, or sufficiently skewed so that 
significant opportunities to develop potential talent are denied to 
certain individuals. As a result, access remains an important 
dimension of performance for st.r.5 systems of higher education. What 
is the structure of attendance at institutions by demographic group or 
by region? What institutions are i>erving what kinds of students and do 
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these patterns imply differences in the types of educational outcomes 
potentially attainable by such groups? What programs do students of 
different type choose to enroll in, and what does this imply about the 
structure of educational access? Such questions have little to do with 
what actually occurs in college classrooms, but may have a great deal 
to do with the ability of a state's higher education system to meet 
established objectives. 

Compared to an assessment of student growth and development, monitoring 
access is a straightforward exercise, and one, indeed, that all states 
already accomplish to a greater or lesser degree. Most such data 
collection, however, currently uses the institution as the level of 
analysis. But access comparisons among institutions tell only part of 
the story. In many cases, the most important patterns are contained in 
statistics on access to programs throughout the system. Examining such 
issues is an important part of monitoring system effectiveness. 

With What Result ? Given that educational gains are achieved in college 
classrooms, and that access to these gains is equitably assured, how do 
these gains translate into actual behavior in the workplace, in the 
economy at large, and in the marketplace and the community? This is 
arguably the most important question of effectiveness from the point of 
view of state policy. 

An initial behavioral question, however, is simply the ability of 
students to successfully complete their programs. Current 
institutional funding mechanisms reward institutions for the volume of 
their instruction, but provide little recognition of its continuity. 
While it is important to recognize that different types of students and 



different types of programs will have legitimately different rates of 
program completion, knowing approximately what these rates are is an 
important system-level responsibility. Many practical problems remain 
in this arena due to the difficulties of accurately assessing program 
completion given initial differences in student intent and program 
structure. But such difficulties can be substantially overcome through 
establishment of common definitions and measurement procedures. 

Assessment of student performance after college is a difficult 
undertaking, but a number of straightforward procedures allow initial 
measurement of these outcomes (Pace, 1979). One important dimension 
here is access to and performance in additional higher educational 
experiences. States which possess a common unit-record data structure 
for their public community colleges and universities have a significant 
advantage in their ability to track individual students through several 
institutions and to assess their performance at different levels. 
Development of such a structure and its associated reporting is an 
important first step in monitoring system effectiveness. Assessing job 
status and performance is more difficult. Federal experience with the 
VEDS program, for example, testifies to problems which may be 
encountered in large-scale student follow-up. Many individual 
institutions, however, successfully track a large proportion of their 
graduates into at least their first job. Their key to success is 
providing sufficient resources to fund a technically sound follow-up 
process (Stevenson, Walleri, and Japely, 1985). A second approach, 
used by at least one state, is to link educational records with 
employment records collected for other purposes. If a mandate for the 
legality of such a procedure can be procured through legislative 
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action, this approach is most promising as a means of assessing program 
effectiveness. 

d. At What Cost ? Cost is as much a dimension of systemic effectiveness as 
the actual levels of performance achieved. From the state's 
perspective, improving the effectiveness of higher education is a 
policy question involving trade-offs among many potential claims on 
resources. Estimates of the costs of both base programs and of 
contemplated improvements are critical in order to assess the marginal 
value of such improvements in the light of other investment 
opportunities. 

Assessments of instructional cost in higher education have a long 
history, and remain a difficult undertaking. At the same time, most 
states currently undertake institutional cost comparisons, and many 
examine the costs of individual instructional programs both within and 
across institutions. When coupled with information on student growth 
and development and actual performance in the marketplace, cost data 
take on an additional dimension. In the absence of performance 
information, the tendency is to use cost data alone as a basis for 
program performance. The message for state policy is to temper cost 
information with its effectiveness counterpart. Both measurements are 
difficult, and both are consequently arror-laden. But to use only one 
because the other is assumed to be flawed represents a considerable 
abdication. 

Mapping out the basic dimensions of effectiveness for higher education from 
the state's perspective is a first step in achieving positive change. Only 
a few such dimensions are actually appropriately monitored at the state 
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level— primarily those having to do with access, cost, and selected areas 
of post-college performance. Most assessment is appropriately handled at 
the institutional level, and it is the role of state policy to encourage 
and reward such effort. The next section examines some of the 
characteristics of successful institution-level assessment and improvement 
programs. 

What Kinds of Improvements Can In Fact Be Made? 

To be effective, institutional assessment and improvement programs, 
particularly in the area of undergraduate instruction, must overcome a 
number of structural obstacles. To encourage the development of such 
programs, it is important for state authorities to understand these 
obstacles more fully. A first obstacle is lack of top administrative 
commitment. Colleges and universities are bewilderingly multi-functional , 
and while undergraduate instruction is clearly a goal of most institutions 
it is rarely accorded visible priority. A second obstacle is fragmentation 
of responsibility. The strong departmental structures of most colleges and 
universities, and a division of labor between "academic" and "student 
service" functions, generally means that responsibility and accountability 
for student success and failure is badly divided. A third obstacle is lack 
of incentive for improvement. Most budgetary allocation within 
institutions proceeds on the basis of teaching volume rather than quality; 
furthermore, in most institutions few mechanisms exist for rewarding 
institutional innovation. A final obstacle is lack of concrete information 
about student learning and development. Assessment of student outcomes, 
while a growing activity on many campuses, is still seen as illegitimate by 
many faculty and as insufficiently precise by many others. More 
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importantly, few mechanisms exist for introducing such information into the 
institutional program planning and decisionmaking process. 

That these obstacles can be overcome is evidenced by the experience of 
several dozen institutions which have experimented with comprehensive 
programs of instructional improvement. Illustrative of the range of such 
campus-level instructional improvement programs are the following: 

• The University of Tennessee, Knoxville . For the past five years, 
the state of Tennessee has maintained a Performance Funding Program 
which rewards institutions for demonstrated performance on five 
established criteria. Three of these criteria are defined in terms 
of student outcomes: student learning in general education, student 
learning in the major field, and student satisfaction with the 
educational experience. In response to the Performance Funding 
Program, the University of Tennessee, Knoxville has developed an 
award-winning comprehensive instructional evaluation program. Key 
elements of the program include administration of the ACT-COMP 
instrument to freshmen and seniors to assess learning gain, senior 
testing of all graduates in their majors, and a regularly 
administered student opinion survey. In many cases, individual 
departments have constructed their own senior tests. Results of 
assessment are broken down to the department level, and are used to 
address needed changes in curriculum. These results also enter the 
university's planning/budgeting system through an established 
program review process. The program's impact has been considerable 
as measured by both actual curriculum change and increased faculty 
involvement with students. 
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• Northeast Missouri State University , For the past fourteen years 
Northeast Missouri State University has conducted a "value-added" 
assessment program to ascertain the degree of learning achieved by 
its students. The initial Intent of the program was to demonstrate 
curricular effectiveness by comparing the results obtained by NMSU 
students with national norms on standardized achievement tests. As 
a result, the University's assessment program is founded upon 
available national tests, including the ACT Assessment and ACT-COMP 
(administered to freshmen and readministered to sophomores) and a 
wide range of GRE field examinations and professional certification 
or professional school entrance tests. The University also conducts 
three major surveys of student opinion. The result of the testing 
program has been to focus considerable campus attention on improving 
the quality of instruction. Student gain as measured by 
standardized tests has improved markedly with changes in the 
curriculum, and as a result the university is now attracting better 
students. As a regional comprehensive university. Northeast 
Missouri is typical of many former public teachers colleges in size 
and program array. Unlike most, however, NMSU has chosen to 
emphasize quality undergraduate instruction as its primary mission. 
There is no question that the assessment program has played a key 
role in the University's success. 

• The State University of New York at Albany is a high-quality 
research and graduate institution with a selective undergraduate 
student body. Like many such institutions, however, few mechanisms 
existed several years ago to effectively address issues of 
undergraduate teaching and learning. Through participation In a 
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national project to make better use of student information, 
SUNY-Albany has for the past two years made information on student 
outcomes a part of its annual departmental planning and budgeting 
process. Together with financial and enrollment data, each 
department is supplied with selected data on graduate and current 
student evaluations of instruction at the departmental level. For 
example, each department gets information on student graduate school 
and job placement, and summaries of student perceptions of the 
effectiveness of the instruction they received in light of what they 
are currently doing in graduate school or the workplace. Results 
have been of considerable value in revising curricula to meet 
unknown needs. 

• Miami-Dade Community College . As a large, urban, multi-clientele 
community college, Miami-Dade is fac?^d with a major problem in 
effectively advising students and monitoring their success. 
Approximately ten years ago, the college began a program of 
competency testing coupled with extensive use of computer support to 
automate the advising and placement process. Currently, students 
are tested on entry to determine their ability to enter particular 
progr^s, and their progress is automatically monitored to ensure 
proper progress and to detect early signs of difficulty. Computers 
are used to generate individual advising profiles that indicate 
which courses a given student is eligible for, and which courses 
would have to be taken to complete a particular program. As a 
result, limited faculty and counselor time can be devoted to real 
interaction with students instead of the usual "audit and sign-off" 
function of most such contacts. The testing and evaluation program 
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also allows early detection of student problems so that appropriate 
interventions can be initiated. Rather than focusing on 
departmental or curricular improvements, Miami-Dade has chosen to 
focus on individual student success. Results of the program, as 
measured both by program completion and learning ga->'n, have been 
considerable. 

St. Petersburg Junior College. The assessment and improvement 
program at this multi-campus urban institution has focused first on 
program and curriculum review. As one of several pilot institutions 
for statewide community college program review, St. Petersburg 
Junior College developed a model process for gathering data on 
program effectiveness in meeting regional educational needs. For 
each program under review, the college undertakes a comprehensive 
survey of regional employers and other "consumers" of the college's 
educational product. This survey is based on interviews with actual 
employers, and reviews both their training needs and their 
assessment of the competence on the job of the College's graduates 
in the field. These data are supplemented with statewide data on 
job placement for graduates of occupational programs and data on 
performance at senior institutions for transfer students. Results 
of this process have led to often substantial changes in program 
structure. 

Alverno College . As a small private women's college, Alverno 
College has been particularly interested in individual student 
development. The assessment program at Alverno is both 
comprehensive and individualized, and has as its objectives 
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providing feedback to individual students on their own progress and 
ensuring that the curriculum is effectively meeting established 
educational goals. As part of the assessment process, Alverno 
administers many standardized tests and established tests of 
psychological and personal development (for example, Kolberg's 
scales for moral development and Perry's approach to assessing 
ethical development). These formal measures are supplemented by an 
extensive interview process which assesses development more 
qualitatively and which provides individual students with feedback 
on their progress. The program is supported by a professional 
assessment center, the activities of which have, in effect, become 
part of the curriculum. The Alverno program is a demonstrated 
success, and because of its focus on individual student learning and 
development, has been seen as a model to emulate by many small 
liberal arts institutions. 

These successful programs, and others like them, share a number of common 
characteristics (Ewell, 1984). First, the most successful programs contain 
an explicit focus on the assessment and improvement of individual 
curricula. The curriculum is the heart of the instructional enterprise, 
and it is important to ensure that impacts on the curriculum are clearly 
identifiable. Many programs fail because assessment and improvement is 
only undertaken for the institution as a whole. It is easy for individual 
faculties and departments to avoid responsibility for student success if 
the entire institution becomes the level of analysis. Addressing this 
issue requires that department-level results are publicly available and are 
visibly used in the decisionmaking process. 
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A second element of successful programs is a proactive commitment to change 
on the part of top administration. In each of the cases cited, top 
administration not only supported the effort verbally, but devoted scarce 
resources to ensure that the job was done properly. In each case the 
visible commitment of resources and support to the assessment process 
served as a signal for mid-level administrators to push the effort, and to 
consistently stimulate faculty to re-evaluate their activities. In many 
cases, commitment is demonstrated by a change in the institutional reward 
structure. Several institutions, for example, have experimented with 
special purpose quality-improvement funds allocated to departments and 
units on a competitive basis. Others have introduced data on performance 
into the established budgetmaking process. 

A final element of successful programs is the use of explicit, 
quantitative, institution-specific data on student performance. Despite 
the many drawbacks of the instruments and data-gathering techniques used, 
these institutions have found explicit data on educational outcomes of 
immense value in addressing needed changes. Prior to the existence of such 
data, needed reforms could be blocked by seemingly legitimate claims of the 
need to study the problem further, or by endless discussions about where a 
particular problem was located or its actual magnitude. Even an 
approximate answer, if arrived at through a concrete measurement procedure, 
often proved enough to get needed reforms under way. 

These characteristics are the hallmarks of an institution successfully 
oriented toward self-assessment and self-improvemen^ . Successful state 
policy, in turn, will consist of practices and incentives designed to 
encourage such behavior. 
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4. Given Examples Such as These. What Can States Actually Do? 



In attempting to effect qualitative improvement in higher education at the 
institutional level, state governments basically have two tools with which 
to work {Jones, 1985). First, they can change the budgetary allocation 
process to provide special funds for needed improvements, or to reward and 
encourage successful performance. In many states, this will mean changes 
in established funding formulas which govern the allocation of dollars to 
individual institutions. In others, it may involve setting aside resources 
to be allocated by means of a separate process. Secondly, states can exert 
control over institutional behavior directly through regulation or statute. 
In some cases this will mean initiating requirements for specific 
performance on the part of currently enrolled students or graduates. In 
others it may mean establishing requirements that existing resources be 
expended in designated ways, or that institutions adopt certain explicit 
evaluation activities. 

a- Approaches Based on Funding Allocation . Probably the most powerful 
single lever with which state governments can seek to influence 
institutional behavior is their method for allocating resources. But 
by its very nature, use of the allocational lever tends to be limited: 
given substantial existing investments and the need to maintain the 
asset, shifts in actual dollar allocations among institutions tend to 
be long-term and marginal. 

Recent reviews of state-level allocation mechanisms indicate that 
approximately half of the states currently allocate funds to 
institutions on the basis of enrollment-driven funding formulas. Such 
formulas tend to have a "levelling" effect on institutions, as the 
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incentives which they provide for are for the same kinds of behavior 
regardless of institution (Brinkman, 1984). Formulas also concentrate 
on input and activity measures— items such as enrollments, faculty, and 
gross square footage—which are readily quantified. Acting in concert, 
these two drawbacks tend to reward quantitative growth over qualitative 
improvement. 

As a result of these drawbacks, many states have been experimenting 
with ways to mitigate the negative effects of formula budgets while 
preserving at least some of the simplicity, clarity and impartiality of 
the formula approach. At the same time, concern for qualitative 
improvement has induced some states to modify the allocation process in 
additional ways. In most cases, this involves setting aside funds for 
particular identified purposes. The most direct use of such funds is 
the one embodied in the Tennessee Peformance Funding program: using 
incentive funds to actually "buy" results at the institutional level. 
The Tennessee Higher Education Commission (THEC) program is almost 
unique in this respect, but some parallels exist in Florida's challenge 
grant program for funding new endowed chairs, and with New York's 
established degree capitation grant program (Bundy Funds) for private 
institutions. 

Most extant state-level improvement programs, in contrast, are 
identifiably special purpose. Tennessee's "Centers of Excellence" 
program provides an example of such an approach. States such as 
Virginia have used set-aside funds to establish grant-like programs 
that foster innovations in curriculum or administration at the 
institutional level. Finally, states like Missouri have experimented 
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with allocating program improvement dollars on the basis of projected 
"returns on investment" in actual learner outcomes, in response to 
individual institutional request. 

As several recent observers have noted {Brinkman, 1984; Spence and 
Weathersby, 1981), such "set-aside" and categorical grant programs have 
the major virtue of allowing enrollment-driven allocation formulas to 
do what they do best— to provide base funding in appropriate amounts to 
maintain fundamental operations and assets— while providing additional, 
targeted incentives for carefully selected qualitative improvement 
areas. Because the focus of such programs is indirect, powerful forces 
are exerted for institutional flexibilty in responding to the 
incentives provided. One of the best examples of how such a program 
can work is provided by the case of the University of Tennessee, 
Knoxville described above. 

But problems with this approach are also apparent. A major difficulty 
lies in negotiating appropriate criteria for distributing special 
purpose funds. The Tennessee solution is so far unique, and arrived at 
its current form only after many iterations between state authorities 
and institutions about what might be feasible. Such discussions will 
take different turns in each state that attempts to set explicit 
criteria of this kind. The grant-like process used by states such as 
Virginia represents an alternative approach, but one which provides no 
hard link between allocation and performance. 

A second difficulty with this approach is that of maintaining an 
appropriate balance between special purpose funds and the base funds 
needed to maintain the institutional asset. Qualitative improvements 
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are by nature marginal— they are added on to existing activities. But 
once qualitative improvement is recognized as a priority, the 
temptation surely exists of treating the entire funding process as 
performance funding. Doing so amounts to reopening the case for 
effectiveness on each aspect of an institution's program each year or 
biennium. As states that experimented with zero-based budgeting a 
decade ago can testify, this is likely to be a frustrating and 
fruitless exercise. 

Regulatory Approaches to Quality Improvement . Rather than changing the 
allocation process, a number of states have opted to directly require 
certain kinds of activities as part of their exercise of regulatory 
authority. A few states have responded to public concern about 
academic quality, for example, by mandating that students demonstrate 
specific levels of performance. In Florida, students in puolic 
institutions completing their sophomore year must successfully pass a 
statewide College Level Academic Skills Test (CLAST) as a condition for 
enrollment with junior standing. New Jersey is embarking upon a 
similar program for implementation next fall. In Mississippi and 
Missouri, moreover, graduates of teacher education programs must score 
above designated levels on a standarized achievement test as a 
condition of graduation. Finally, in South Dakota, the public higher 
education system is currently implementing a requirement that all 
students be tested in both "value-added" terms and in comparison with 
national norms for performance in their major fields. 

Even where explicit performance requirements have not been put in 
place, states have used their regulatory authority to induce 
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qualitative improvement. For example, there is a recent and rising 
trend tjward legislative and executive audits of public colleges and 
universities to be conducted as performance audits. Traditionally, 
such audits have been almost exclusively confined to financial 
regulation— ensuring that institutions spent monies the way they said 
they would. A second example is provided by institutional and 
statewide program review processes. As indicated by a recent survey of 
state higher education agencies, more than two-thirds of the states now 
have explicit academic program review processes in place, and most of 
these are of recent origin (Barak, 1982). Such reviews are now 
increasingly performance-oriented, and require that the program provide 
data about student performance and subsequent educational and job 
success. For example, in Hawaii a new statewide program review process 
for community colleges is currently being pilot tested that includes 
substantial data on student placement and performance, 

A third mechanism, more indirect, is to require institutions to produce 
data on outcomes as part of their statistical reporting obligations. 
State boards already require that public institutions report consistent 
data on enrollment and finance each year. Even if funding levels or 
programs- remain unaffected, the act of requiring such data can signal 
concern about the issue to campuses, and can begin the process of 
change. In North Carolina, for example, university system liUthorities 
have for a number of years required that campuses submit data on 
student progress through the curriculum, and as a result, all public 
North Carolina institutions now possess longitudinal student tracking 
systems. The simple presence of such data on actual student progress. 
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unavailable at most institutions in other states, has stimulated 
changes in curriculum, and in placement and advisement. 

These approaches share a number of common benefits. First, use of the 
state's regulatory authority allows direct communication of what is 
wanted and intended. Because finance mechanisms tend to be indirect, 
institutitions can sometimes find ways to meet the requirements for 
incentive funding without making substantial changes. Regulatory 
mechanisms, in contrast, particularly those based on actual student 
testing, are directly focused on the outcome to be produced. For 
example, Florida's "Gordon Rule" that requires freshmen to write a 
total of 50,000 words in the course of their initial year has indeed 
induced Florida campuses to address the issue of writing. Such 
mechanisms are at their best when they can be directly focused on a 
particular issue of this kind. 

This strength, however, becomes a potential weakness when more general 
improvements are sought. Statewide testing programs (particularly such 
initiatives as the "rising junior" test programs in Florida and New 
Jersey) can involve considerable potential danger of 
overstandardization, and may induce all institutions to produce similar 
products. For example, in the "value-added" portion of Tennessee's 
Performance Funding program, questions have arisen about the 
appropriateness of comparing gain scores from the University of 
Tennessee, Knoxville — a selective campus whose students start 
relatively high on normed tests such as the ACT-COMP— wi th other less 
selective campuses. Indeed, the impact on the institutional admissions 
process of all testing programs should always be carefully considered 




by state authorities. Performance testing will induce institutions to 
limit access based on initial ability; "value-added" testing will have 
the opposite effect. 

In addition, there is a substantial danger involved in using single 
indicators of institutional performance. Testing programs, by their 
very nature, tend to direct attention to the test itself rather than 
toward what is being measured. So long as the tests chosen actually 
match state concern, this is not a problem; "teaching to the test," 
despite its odious sound, may be exactly what is wanted. But care 
should be taken to ensure that institutions are assessed on other 
performance criteria as well. 

Because of these difficulties, indirect regulatory mechanisms such as 
program review have considerable appeal. First, they are already 
familiar to institutions, as more than two-thirds of the states already 
have such processes in place. Moreover, most such processes resemble 
specialized accreditation with which many programs are already 
familiar. Secondly, program review processes are already focused where 
they should be in order to achieve meaningful change— on the 
curriculum. -Thirdly, their structure demands considerable 
participation from those who really need to be involved in the change 
process~~f acuity and mid-level administrators. Finally, program review 
processes are multi-dimensional and thus avoid the dangers of 
single-indicator systems. They allow for the fact that differences 
will exist among institutions and programs with regard to both intent 
and clientele. 
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Seme Directions for State Policy 

All of the above mechanisms are -available to state governments to begin the 
process of inducing change at the campus level. Experience with them so 
far has been limited, but suggests a number of broad directions for state 
policy. Among them are the following: 

• Monitor the "big picture ." The appropriate focus for state-level 
assessment and data collection is the effectiveness of the system, 
not the details of institutional functioning. As a result, state 
data collection should concentrate on systemic questions such as 
access, cost, and student placement/performance in productive roles. 
At the same time, institutions themselves should be induced to 
undertake local assessment of their own effectiveness in teaching 
and learning. 

• Recognize that institutional diversity is healthy, and should be 
preserved, so long as statewide educational goals are being 
attained. Considerable research has shown that the most effective 
colleges and universities are those with explicitly focused, 
distinct, institutional missions. State policy should be carefully 
fashtoned to avoid assessment or regulatory policies which might 
homogenize important institutional differences, and thus dilute 
overall effectiveness. 

• Create positive incentives for institutional improvement . A key 
point of successful programs for change at the institutional level 
is that they are based on the notion of reward for appropriate 
initiative rather than on the basis of punishment through regulation 
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or statute. Certainly a balance between the two needs to be 
maintained, and the best programs will contain elements of both. 
But experience suggests that the overall tone of an improvement 
program be one based on reward. 

• Visibly di stinguish incentive structures for qualitative improvement 
from regular institutional funding mechanisms . Maintaining a 
distinction between "base" and "improvement" funding provides a way 
of avoiding considerable institutional resistance to change. So 
long as qualitative reallocation remains marginal, institutions will 
feel less threatened about the ultimate impact and direction of such 
programs. Experience suggests that such incentive funds probably be 
limited to less than ten percent of total allocation. 

• Leave institutions with considerable discretionary authority on how 
to accomplish quality improvement goals . The focus of state 
programs should be placed upon setting appropriate goals and on 
monitoring systemic performance. Individual institutions should be 
held accountable for (1) demonstrating through whatever data are 
institutionally appropriate that they are in fact producing a viable 
educational product consistent with their assigned mission, and (2) 
demonstrating that they have in place self-assessment and 
self-corrective mechanisms for dealing with detected problems 

local ly. 

• Stress the use of concrete, quantitative information on 
institutional and system performance . The key to successful systems 
at the institutional level is that they are based on the collection 
and public display of concrete data. Using explicit assessment 
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measures has many dangers, but has the substantial advantage of 
providing clear signals on what is considered important. Concrete 
data also allows problem areas to be immediately identified and 
addressed. With regard to data collection, Tennessee's motto is 
appropriate, "act on the possible while awaiting perfection." 

• Use multiple indicators of institutional and system performance . 
While concrete assessment information is important, single 
indicators of performance should be avoided at both the state and 
institutional levels. Concentrating on only one or two indicators 
of performance creates powerful incentives for institutions to 
manipulate the indicators rather than to actually improve 
performance. 

• Wherever p ossible, use existing information . Institutions and 
states already collect considerable information about student 
outcomes and performance. A key aspect of successful institutional 
assessment programs has been to encourage aggregation and use of 
such data for planning and review purposes. State policies should 
provide mechanisms for aggregation of this kind, for example, 
through program review processes. Such mechanisms should seek a 
wide (but not necessarily standard) array of data from institutions, 
and Should emphasize the use of such information at the local level. 

These general directions are intended only as a starting point for state policy 
development. Individual states will vary considerably in their intended 
outcomes, in the policy tools which they have available to achieve change, and 
in the array of institutions which comprise their higher education systems. In 
devising quality improvement programs, however, all states will do well to 
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remember that colleges and universities possess a considerable capacity to 
Identify their own problems and to act in a creative manner to solve them, 
best state programs will be the ones which effectively recognize and harness 
these forces. 
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